Exchange of data between an external data source and an integrated medical data display system

ABSTRACT

A method for exchanging data between an external data source for annotations and an integrated medical data display system, comprises: determining information displayed on a screen of the integrated medical data display system by capturing the screen; and performing at least one of (i) selecting data from the external data source assigned to the determined information and displaying the selected data complementary, or (ii) extracting annotations from the integrated medical data display system based on the determined information and appending the extracted annotations to the external data source for annotations.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application claims priority under 35 U.S.C. § 119 toEuropean Patent Application No. EP 21200319.8, filed Sep. 30, 2021, theentire contents of which are incorporated herein by reference.

FIELD

One or more example embodiments of the present invention relate to amethod for exchanging data between an external data source forannotations and an integrated medical data display system. One or moreexample embodiments of the present invention also concern a dataexchange device for exchanging data between an external data source andan integrated medical data display system. Further, One or more exampleembodiments of the present invention relate to an AI-based data analysissystem.

BACKGROUND

AI-based applications (AI=artificial intelligence) are more and morewidespread in the field of medical assessment, in particular, formedical images (e.g., magnet resonance images, computed tomographyimages, X-ray images, Single photon-emission computed tomography images,positron emission tomography images, or ultrasound images). However, abroad adoption is currently hindered by IT problems and in particularintegration problems. Especially, the lack of standard mechanisms forthe display of AI-results and for interacting with them in the PACS(PACS=Picture Archiving and Communication System) and/or the RIS(RIS=Radiology Information System) pushed AI vendors to integrateAI-based applications into PACS and/or RIS using proprietary interfacecontracts or using their own application typically in an additionalapplication, e.g., a web browser, also called “review screen” or“confirmation screen” or “confirmation UI” (UI=user interface). Thislast approach presents the challenge of an additional window that mostof the time does not behave synchronously with the main PACS-viewer. Theuser has to perform actions twice. For example, if he scrolls to aparticular slice in the PACS viewer, he will also have to redo thisoperation in the AI confirmation screen. These approaches are notconvenient for a proper and effort-less integration of artificialintelligence in radiology and decrease the broad usage of artificialintelligence: It is now well-known that superior routine workflowintegration is a key for AI success in medical imaging.

One conventional approach is to develop a product which encompasses aviewer and an AI-tool. A tool comprises a software which enables acomputer-aided application of a method, preferably in dialogueoperation. Another approach is directed to establish compatibilitybetween both components through a proprietary API (API=ApplicationProgramming Interface). General interoperability is guaranteed throughestablished standards and real-time data exchange between the twoapplications, deep integration via an iFrame-embedded approach orsimilar.

SUMMARY

In reality, there is a multitude of PACS and AV (AV=Advancedvisualization) viewers in use at clinical sites, making it fairlyimpractical and cost-prohibitive to establish a common interface ofdeeper integration with every single medical viewer, and especiallyimpractical to use an external application window, also mentioned asconfirmation UI, in conjunction with an existing PACS viewer, since bothare completely desynchronized.

The inventors have determined that it would be appropriate to develop asolution, which would facilitate the integration and display of suchexternal AI results onto any PACS viewer by making them available in thecorrect context, i.e. image slice and potentially well-integrated in themain viewing environment, i.e. overlay.

The inventors have also identified a further problem occurring in theabove-mentioned scenario, which is the challenge of an additional windowthat most of the time does not fit well in a chosen default PACS/RISlayout or a layout of an AV viewer, as it hides useful contents. Forexample, most of the time, the additional window is displayed on thefull screen and has to be moved manually, if the position and size isnot suitable. Similarly, smaller tools like external toolbars or widgetsare usually auto-positioned only at some fixed locations, e.g. thebottom right corner. These approaches are not convenient for a properand effort-less integration of artificial intelligence in radiology.

As mentioned-above, an approach is to have the viewer and the AI-tool tobelong to the same product. Compatibility between both components can beestablished through a proprietary API, general interoperability can bealso guaranteed through established standards, deep integration via aniFrame-embedded approach or similar. However, as also mentioned-above,in reality, there is a multitude of PACS and AV viewers in use atclinical sites, making it fairly impractical and cost-prohibitive toestablish a common interface or deeper integration with every singlemedical viewer, and especially impractical to embed an externalapplication UI (UI=user interface) into an existing PACS/RIS UI(RIS=Radiology Information System) for example, since those approachesare costly and done in very rare cases.

An example for AI-based medical information is the detection andquantification of lesions that bears great potential to reduce readingtime of radiological images. This can be achieved by automating awayrepetitive and manually tedious tasks such as diameter measurements ofpulmonary nodules. Given the imperfect performance of AI algorithms,there is however always the need to account for user-added lesions thatwould have to be tracked as well on follow-up scans. Generally, theuser-added lesions are added in the PACS/RIS UI, whereas theconfirmation UI is typically not able to access to the user-addedlesions. If the software were not able of such functionality, this wouldcome at larger workflow costs. Size sorting between the set of AI-foundnodules and those sets with user-added nodules would be inconsistent andtarget lesions could be different. In such case, the user would have toswitch between different tools and take into account AI-found lesionsand separately user-added lesions. All workflow improvement would begone in such a case. It is therefore of utmost importance to captureuser-added lesions.

Also the last stated special problem is generally solvable if a viewerand an AI-tool, i.e. a confirmation UI, belong to the same product orcompatibility between both component is established through aproprietary API or general interoperability is guaranteed throughestablished standards, or similar. In reality however, there is amultitude of PACS and AV viewers in use at clinical sites, making itfairly impractical and cost prohibitive to establish a common interfacewith every single medical viewer.

Hence, a general problem underlying one or more example embodiments ofthe present invention is to improve the exchange of data between anexternal data source for determination of annotation data, in particularan image processing application, and an integrated medical data displaysystem like a PACS-System.

The before-mentioned problem is solved by a method for exchanging databetween an external data source for annotations and an integratedmedical data display system, an exchange device for exchanging databetween an external data source and an integrated medical data displaysystem, and/or a medical data analysis system according to one or moreexample embodiments of the present invention.

According to the method for exchanging data between an external datasource for annotations, which external data source works preferablyAI-based, and an integrated medical data display system, for example aPACS workstation, information, which is displayed on a screen of theintegrated medical data display system is determined by capturing thescreen. As later discussed in detail, capturing the screen can beimplemented by screen capture techniques.

Screen capture techniques can be realized using a so-called screencamera, which comprises a software which generates a digital image orvideo from a computer screen.

In this context, it is preferred that the external data source forannotation data comprises a preferably AI-based, but not limitedthereto, image processing application and its database. As will also bediscussed in detail later, the information displayed on the screen ofthe integrated medical data display system can encompass medical dataas, for example, medical images and/or annotations marked by a user. Theinformation can also incorporate formal information, especially layoutinformation, which can be used for finding out an appropriate region onthe screen for positioning a window for an external application, forexample a user interface window. Further, at least one of the followingsteps is performed: Data from the external data source assigned to thedetermined information are automatically selected and the selected dataare displayed complementary to the displayed information. That meansthat the selected data are displayed on positions such that the selecteddata do not conceal the displayed information. Alternatively oradditionally, annotations, preferably annotations added by a user, areextracted from the integrated medical data display system based on thedetermined information, and the extracted annotations are appended tothe external data source for annotation data.

Advantageously the method does not need a strict integration betweendifferent systems, which would be not appropriate due to the pluralityof different display systems and data source systems.

A tool, for example implemented as a data exchange device, for analysingthe retrieved information, preferably runs on-premises, i.e. on thehardware running the integrated medical data display system, for examplea workstation, itself.

The method also ensures optimal display of content of external datasources in a confirmation UI, even in third-party PACS/RIS environmentsand an “embedded look and feel” without manual configuration.Information can be used for a multitude of products of digital healthsolutions that need to be used in conjunction with other systems, forexample viewers or worklists, mainly all products containing a viewerthat could benefit in being called up and used in context, such as asystem focused on AI-based medical image processing, or a system focusedon AI-based decision support for clinical pathways.

Further, the method also ensures optimal display of user interfacewindows, even in third-party PACS/RIS environments and “embedded lookand feel” without manual configuration or complex integration tasks. Themethod can be used for a multitude of products at digital healthapplications that need to be used in conjunction with other third-partysystems, for example viewers or worklists like AI widgets, result reviewscreens from AI- or post-processing (interactive) applications, etc.

The additional step of extracting annotations from the integratedmedical data display system based on the determined information and ofappending the detected annotations to the external data source forannotations in combination with an optional step of actively re-trainingAI-based algorithms for determining (new) annotations, improves ortailors the algorithm performance, for example for the detection oflesions. Further, the method can be used to track suspicious objects orphenomena like lesions over time.

The data exchange device for exchanging data between an external datasource for annotation data and an integrated medical data displaysystem, for example a PACS server, according to one or more exampleembodiments of the present invention, comprises a determination unit fordetermining information displayed on a screen of the integrated medicaldata display system by tracking the screen and at least one of thefollowing units: a selection and display unit for selecting data fromthe external data source assigned to the determined information anddisplaying the selected data complementary to the displayed information.The selected data are displayed such that the selected data do notconceal the displayed information to the displayed information.Alternatively or additionally, the data exchange device can alsocomprise an extraction unit for extracting annotations from theintegrated medical data display system based on the determinedinformation and appending the detected annotations to the external datasource for annotation data. The data exchange device shares theadvantages of the method for exchanging data between an external datasource for annotation data and an integrated medical data display systemaccording to one or more example embodiments of the present invention.

The preferably AI-based medical data analysis system according to one ormore example embodiments of the present invention comprises a remoteexternal data source for storing medical image data, a preferablyAI-based evaluation unit for evaluating or processing medical imagedata, i.e. preferably the external data source for annotation data, anintegrated medical data display system for displaying evaluated medicalimage data and a data exchange device according to one or more exampleembodiments of the present invention. The medical data display systemmight comprise a third-party PACS system. The medical data analysissystem according to one or more example embodiments of the presentinvention shares the advantages of the method for exchanging databetween an external data source for annotation data and an integratedmedical data display system according to one or more example embodimentsof the present invention.

The essential components of the data exchange device according to one ormore example embodiments of the present invention can for the most partbe designed in the form of software components. This applies inparticular to the determination unit, the selection and display unit andthe extraction unit of the data exchange device, but also parts of theinput interfaces. In principle, however, some of these components canalso be implemented in the form of software-supported hardware, forexample processors or FPGAs or the like, especially when it comes toparticularly fast calculations. Likewise, the required interfaces, forexample if it is only a matter of transferring data from other softwarecomponents, can be designed as software interfaces. However, they canalso be designed as hardware-based interfaces that are controlled bysuitable software. Furthermore, some parts of the above-mentionedcomponents may be distributed and stored in a local or regional orglobal network or a combination of a network and software, in particulara cloud system.

A largely software-based implementation has the advantage that medicaldata analysis systems that have already been used, can easily beretrofitted by a software update in order to work in the manneraccording to one or more example embodiments of the present invention.In this respect, the object is also achieved by a corresponding computerprogram product with a computer program that can be loaded directly intoa memory device of a medical data analysis system, with programsections, in order to carry out all steps of the method according to oneor more example embodiments of the present invention, if the program isexecuted in the medical data analysis system. In addition to thecomputer program, such a computer program product may contain additionalcomponents such as a documentation and/or additional components,including hardware components such as hardware keys (dongles etc.) forusing the software.

For transport to the medical data analysis system and/or for storage onor in the medical data analysis system, a computer-readable medium, forexample a memory stick, a hard disk or some other transportable orpermanently installed data carrier is used on which the program sectionsof the computer program that can be read in and executed by a computerunit of the medical data analysis system are stored. The computer unitcan comprise for example, one or more cooperating microprocessors or thelike used for this purpose.

The dependent claims and the following description each containparticularly advantageous embodiments and developments of the presentinvention. In particular, the claims of one claim category can also befurther developed analogously to the dependent claims of another claimcategory. In addition, within the scope of the present invention, thevarious features of different exemplary embodiments and claims can alsobe combined to form new exemplary embodiments.

In a variant of the method according to one or more example embodimentsof the present invention the determined information displayed on thescreen comprises at least one of the following types of information:

-   -   content of medical data and/or    -   formal information about a format of displayed data,    -   non-image information, comprising patient metadata or        examination metadata.

Content of medical data comprises information describing the subjectmatter of medical data. Medical data comprise patient data andidentification data of the treating doctor. Patient data comprise allpersonal information related to a patient, determined or acquired orprocessed or stored in a medical institution.

The format of displayed data comprises information about the shape andsize of displayed structures, in particular the windows and sub-windowsof the screen.

Patient metadata or examination metadata comprise attributes, used fortagging and categorizing patient data or examination data for fast andefficient searching processes. Patient metadata comprise, for example,an identifier of a patient. Examination metadata, for example, comprisean identifier of a study.

In case the information displayed on the screen comprises content ofmedical data, for example medical image data, the image data can bespecified and personalized. Further, specific annotation data related tothe specified and personalized image data, for example annotationsrelated thereto, can be retrieved from the external data source and canbe displayed in combination with the medical images, which are currentlydisplayed.

In case the medical data comprise annotations, the annotations can beextracted using the method according to one or more example embodimentsof the present invention and added to an external data source, whichgenerates annotations related to the same medical images usingartificial intelligence. Annotations comprise notations added to medicalimage data. Such notations can comprise labels of findings in medicalimage data (e.g., the location and/or the classification of certainnodules, wherein classification can, e.g., correspond to a level ofmalignancy or benignancy of the respective nodule) or segmentations ofmedical image data (e.g., where organs or parts of organs are identifiedwithin the medical image date, e.g., for radiation therapy planning orfor volumetric measurements).

In an optional step of the method according to one or more exampleembodiments of the present invention user made annotations and theautomatically generated annotations are fused in the external datasource. Hence, an analysis of medical images is improved. In a furtheroptional step of the method according to one or more example embodimentsof the present invention, the user added annotations are used for(re-)training the preferably AI-based generation of annotations suchthat the automated analysis of the medical images is improved.

In a variant of the method according to one or more example embodimentsof the present invention the determined information displayed on thescreen comprises formal information about a format of displayed data,and the formal information comprises a screen layout comprising a screenportion, the screen portion including at least one of a viewport and adelimited screen portion including an image segment; and/or the formalinformation comprises an user input in the integrated medical datadisplay system.

In case the determined information displayed on the screen comprisesformal information, for example information about a screen layout, asfor example the locations and borders of displayed viewports, activeviewports can be detected and monitor segments of least-interest, i.e.non-active viewports or patient information, that has been looked atalready by the user or that is duplicated somewhere else, can bedetermined. These segments can be used for displaying an externalapplication window, e.g. a widget, a notifier or a confirmation UI. Theexternal application window can be used as user interface forinteracting with the external data source on a screen of the integratedmedical data display system. The formal information, i.e. formal screencontent, can also comprise user input in an image viewer, which may givea hint for a preferably used region on a screen, e.g. regions of “highattention” that might be well-suited to display content which needsvisibility.

Non-image information comprising patient metadata or examinationmetadata may include contextual content from RIS/reporting screens, forexample a worklist, a patient's history, etc.

In a further variant of the method according to one or more exampleembodiments of the present invention, the step of determininginformation comprises at least one of the following sub-steps:

-   -   using a tool capable of screen tracking to capture screen        content preferably in real-time, the tool can be used for        capturing screen layout and/or user input,    -   detecting, preferably by OCR (OCR=optical character        recognition), which study or patient is currently displayed on        the screen, for example by monitoring a particular region of the        captured screen layout, in which patient ID or an accession        number is displayed, or looking for such a data point in the        captured screen layout,    -   classifying based on the captured screen layout, which screen        portions correspond to which image segments, for example by        determining (x,y)-coordinates of bounding boxes defining        viewports and, for each classified image segments, identifying        which image the user is currently visualizing and is eventually        active on, either by analysing the metadata displayed in the        corners of the corresponding viewport or by cross correlating        the pixel data with a set of images of this patient known by an        external application, like the external data source for        annotation data, in that context, also the slice number and        series of the displayed image can be determined for        identification,    -   identifying which window level the user is working with for a        particular slice, optionally in order to infer what organs he        might be looking at. For example, the lung window, typically W:        1500 L: −600, e.g. grey values centered at −600 HU        (HU=Hounsfield units) and with a window width of 1500 HU, would        be indicative of the interest of the radiologist in the lungs        and not in other organs.

In particular, in the case the step of determining information comprisesthe substep of classifying based on captured screen layout, which screenportions correspond to which image segments, the determined informationdisplayed on the screen comprises formal information about a format ofdisplayed data, wherein the formal information comprises a screen layoutcomprising a screen portion, wherein the screen portion comprise atleast one delimited screen portion including an image segment.

In this context it has to be mentioned that a screen can be divided in aplurality of delimited screen portions. These delimited screen portionscan comprise bounding boxes or viewports as limiting frame. Each ofthese screen portions can be realised as an image segment, which depictsa special type of images. In particular, an image segment comprises orrelates to at least a part of a medical imaging dataset being reviewedby a user. For example, if a three-dimensional DICOM study is to bedisplayed, an image segment can correspond to a certain two-dimensionalcutting plane through the three-dimensional study, or to certainprojection of the three-dimensional study (e.g., a maximum-intensityprojection), or to a certain rendering result of the three-dimensionalstudy.

OCR, i.e. optical character recognition comprises an electronicconversion of images of printed text into machine-encoded text. OCR canbe used for recognizing text on the screen. In this context, OCR can beused to recognise keyword related to information about patient ID or anaccession number.

Windowing is the process in which a CT image greyscale component of animage is manipulated via the CT numbers, i.e. the HU values. Thebrightness of the image is adjusted via the window level. The contrastis adjusted via the window width. The window width is the measure of therange of CT numbers that an image contains.

Generally, the identification of the window level the user is workingenables the use of the same window level also in the confirmation UI.

It is envisioned that these operations, for example the correlations,would need to happen preferably in the cloud or in an edge server, butnot in workstation, the user is currently using, since these remotesystems are preferably the locations having the missing part that has tobe correlated: for example, the PACS/workstation shows a slice, and thecloud/edge possesses results for this slice. This implies that the datais transmitted using data-privacy-preserving methods (e.g., pseudomizingthe image by replacing identifying metadata like the name of the patientwith non-identifying metadata like an identifier number or string thatcannot be associated with the patient by the remote system). Upload anddownload speeds at the site might also be a decisive factor in thischoice.

In a further variant of the method according to one or more exampleembodiments of the present invention, the external data source comprisesresults, preferably AI results, and the step of selecting the data fromthe external data source comprises searching the external data sourcefor results which correspond to the determined content and/or filteringthe data available results in the external data source, wherein theselected data comprise the results. That means that it is, for example,queried for results for the image series and slice, identified by screencapture on the screen of the integrated medical data display system. Inparticular, the results, preferably AI results, are results related tothe medical images that can be accessed using the workstation.

In a variant of the method according to one or more example embodimentsof the present invention, the step of displaying selected data comprisesgenerating an overlay comprising annotations corresponding to thedetermined information on top of the image (slice) shown on the screenof the integrated medical data display system, but independently of theintegrated medical data display system.

The information displayed on the screen of the integrated medical datadisplay system, for example medical image data, can be understood as afirst application window. Then the overlay can be realised as a secondapplication window, which is at least partly transparent. The overlaycomprises a second application window which is presented in foregroundoverlapping with the displayed information of the integrated medicaldata display system. The overlay is registered with the determinedinformation, for example a medical image of the PACS, such that theresults depicted by the overlay, for example annotations, are located atthe corresponding positions of the displayed information. In particularusing a registration enables to adapt the position and size of theoverlay to the underlying displayed information. In particular, theoverlay is spatially registered with the determined information, e.g.,by adapting the position and/or the size of the overlay to match thedisplayed information.

The overlay can be generated in a way to maximize usability usingappropriate colours or styles that make the result well distinguishablefrom the original image. If the overlay is interactive, then it needs tohave the focus. While scrolling on the screen of the integrated medicaldata display system, for example the PACS, it would potentially alsoneed to have the focus. This might be solvable by adapting the size ofthe overlay so that scrolling in the background application is enabled,or by implementing an event-forwarding logic to the background program.

Alternatively, an overlapping of the selected data at least partiallywith the images of the integrated medical data display system can begenerated, if no sufficiently suitable region of least interest isfound.

In a further variant of the method according to one or more exampleembodiments of the present invention, the step of displaying selecteddata comprises showing the selected data in a particular external windowthat is opened next to the integrated medical data display system. Theexternal window can comprise a result review and visualization screen ofan image post-processing application, for example an AI-based imageanalysis application. This window, similarly to other programs found oncomputer, might always stay in the foreground and could only be broughtback to background or be hidden by an additional button click or toggle.

For example, results shown in this external window can be presented in a“mini viewer” showing results computed by the AI application, or in a“widget” showing list-like textual and iconographic information. Forexample, if the radiologist is looking at a chest CT in a particularlung window, i.e. with window level and width identified earlier, theapplication can present only results related to the lungs, e.g. airways,nodules, parenchyma and omit results related to bones, heart and aorta.If the method is obtained at sufficient speed/performance, i.e.near-real-time, the method can offer a synchronized scrolling-likefunctionality for an external application, which is able to follow theimages that a user is looking at on the screen of the integrated medicaldata display system, for example a PACS. If performance is not met toprovide results for the exact same slice that the radiologist is lookingat, for example because he is scrolling fast, an inference could stillbe made for the next findings to show considering, e.g. in whichdirection the radiologist scrolls through the images. For example, thetool is prepared to show the findings of slice 270, if the radiologisthas scrolled through the slices 267-268-269 already.

Overall, the “near real-time” behaviour might be achievable only byusing additional techniques, e.g., to reduce the dimensionality of theimage without losing information (lossless compression), for example bydown sampling or using an autoencoder-based approach. The streaming ofthe data in general has to be technically optimized, which might resultin using only an (on-premises) edge server and not the cloud.

In a further variant of the method according to one or more exampleembodiments of the present invention, the information displayed on thescreen comprises the current layout of reading workplace monitors. Thatmeans the information, how many viewports and where on PACS monitorsthey are positioned and where are metadata shown outside of theviewports like RIS/reporting contents, eventually located on separatenon-diagnostic monitors. Such layout information can be retrieved fromthe determined information by using image processing techniques as laterdiscussed in details. That information can be used for finding out theregion on a set of monitors, which is of least interest and can befurther used for displaying an interface for interacting with anexternal data source. In another aspect of one or more exampleembodiments of the present invention, this can be used to detect regionsof highest interest in order to use them to show important content(e.g., to increase the probability of a time-critical information to beseen by the user), or on the contrary to preserve them from any hidingby external windows.

Furthermore, in a variant of the method according to one or more exampleembodiments of the present invention, the step of determininginformation comprises determining a corner or locations or borders ofthe displayed viewports based on the captured layout. This can be donevia a learning-based approach, which nowadays would give very goodperformance as the task is quite easy. Another alternative would bestandard image processing techniques like convolutional processing.There are for example well-known specific kernels that enable thedetection of horizontal and vertical lines of different width, whichwould then enable, by thresholding, to retrieve the different viewports.Line detection has been a quite widely researched image processing topicin the past, which is also well mastered. Finally, we could also foreseea configuration setting or some fixed rules. Though this would probablyallow for less flexibility in the kinds of layouts the system can dealwith. For this to work, one idea could be to use a first screenshot thatcaptures the overall structure of the used screens and then to requestthe user, using some user interface, to define regions on this imagethat could be considered “least interest” or viewports for example. Thiscould be done for a set of common layouts that would then be persisted.

In a variant of the method according to one or more example embodimentsof the present invention which is related to the version for finding aregion on a display of an integrated medical display system the step ofdetermining information comprises:

-   -   determining one or more active viewports using mouse activity        tracked using screen capture, or    -   determining screen portions not containing image segments,    -   displaying an external application window within the determined        screen portion.

These steps can be carried out for finding portions or regions on ascreen, which are currently not focused by a user and can therefore beused for positioning a window for an interface between user and externaldata source.

In a further variant of the method according to one or more exampleembodiments of the present invention, the step of determininginformation comprises using a metric to determine at least one screenportion as a “monitor segment of least-interest” based on the determinedinformation. Such a segment can comprise non-active viewports or patientinformation that has been looked at already or that is duplicatedsomewhere else. For example, patient information that has been looked atalready, are provided with a tag or are displayed in a smaller windowfor an unambiguous identification of that type of patient information.The variant also comprises the step of displaying an externalapplication window within the determined screen portion.

The metric can be defined such that a small monitor segment is not asinteresting as a more extended monitor segment.

However, such a metric can comprise much more complex criteria. Thesecriteria may comprise:

A monitor segment is not interesting in one of the following cases:

-   -   it contains only text that is very small,    -   it contains only text that is better visible elsewhere on the        screens, because it is displayed there bigger or more centrally,        for example,    -   it contains only small images, also called thumbnails, and all        of the images are displayed in large version somewhere else on        the screen(s).

One of the segments of least-interest, for example a non-active viewportor a non-image segment, can be reserved for displaying at this locationan external application window e.g. a widget, a notifier or a resultreview UI which has potentially been properly resized to fit this spaceand display correctly for interaction of the user. This way, the userhas the feeling that the external application is properly integrated andis enabled to watch all relevant parts of the original images, since theexternal application does not hide vital portions of the originalimages.

The successful resizing can be achieved easily with a responsive designof the application and with a resizing of a container window. Acontainer window is an external application window or the box in whichit is integrated or the space the external application window takes in ascreen.

In another variant of the method according to one or more exampleembodiments of the present invention, in case several candidatepositions are available, the optimal position can be chosen based on atleast one of the following techniques:

-   -   based on best size fit between the external application window        and the monitor segments of least-interest,    -   based on aspect ratio fit between the external application        window and the monitor segments of least-interest,    -   overlapping the integrated medical data display system if no        monitor segment of least interest that fits the external        application window is found.

In particular, a certain monitor segment of least interest does not fitthe external application window if the size (width and/or height) of thecertain monitor segment is smaller than the size (width and/or height)necessary for displaying the external application window.

The first technique is related to the amount of the area or length orwidth of a monitor segment, for example a window or viewport, which maybe selected for displaying an external application window.

The second technique concerns the 2D shape of the window selected todisplay the external application window.

The technique of overlapping can be used in case there is not enoughplace for displaying the external application window. If the region ofleast interest is too small, it would be probably better to display thecontent somewhere else. Hence, content will be displayed bigger, whichis an advantage, even if we hide partially middle-interesting content.

The screen capture tool can run on-premises i.e. on the user'sworkstation. In envisioned scenarios in digital health processes, it isassumed that all communications between this tool and the cloud would berouted through a single port and gateway, because hospitals generally donot like opening a direct internet connection to the cloud from theworkstation. However, if tolerated, a simple API call, i.e. from aworkstation to the cloud, with direct internet connection, could beenvisioned. Initiating an interaction directly from the cloud or theedge server would also be feasible.

The screen capture tool mentioned can be implemented by using a standardlibrary. These libraries typically capture the screen as an image oreventually a video and would allow to stream it “continuously” to thedestination of choice. An open-source library like Phantom JS, extendedto screens and not only to webpages, could work. Another option would beto rely on something very standardized like the “screen capture API” forwhich the user would have to “accept” which part of the screen he wouldlike use to monitor, maybe once per session.

Data privacy could be a major concern for this method. It would beprobably necessary to apply data-privacy preserving techniques to modifythe captured images in order to transmit them to the cloud. For example,a file comprising the captured images is encrypted, before transmittingthe file to the cloud and access is restricted to authorized instances,which have a key for decrypting the file comprising the captured images.

The “external application window” could always stay in the foregroundand could maybe only be brought back to background or be hidden by anadditional button click or toggle that it carries. This way, it wouldnot hinder interaction with the background application, while alsoenabling interaction in the “external application window”.

In a further variant of the method according to one or more exampleembodiments of the present invention, the step of determininginformation displayed on a screen of the integrated medical data displaysystem comprises:

-   -   detecting a reference frame based on the information,    -   detecting annotations in the reference frame on the determined        information.

That means in detail for example the use of a tool capable of screentracking to capture screen content in real-time. For identifying areference frame, a frame of reference is preconfigured and autodetected.The reading workflow is usually sufficiently standardized topreconfigure a particular window on a split screen that is always usedto annotate series. Alternatively, a particular window could also beautodetected by for example analysing displayed metadata like seriesdescriptions, which are usually displayed as overlays on the images.Further, user markings, typically in a non-white/non-black colour, canbe detected based on rgb values (rgb=red green blue). Alternatively,image content can be subtracted from unannotated images to compute amask of the annotation. Then the position of the detected user markings,i.e. annotations, is determined. Optionally the detected user markingscan be sent to remote location and new lesions based on the locations ofuser markings can be detected by a preferably AI-based processing unit.In that context, new lesions can be automatically detected using theuser-added lesions as seed points.

In a special variant of the method according to one or more exampleembodiments of the present invention, the step of determining positionof the detected user markings comprises the determination of a slicenumber of the detected user marking. That can be realized by detectingdisplayed metadata using, for example OCR, or by cross-correlating agiven slice with every slice of an unannotated image stack.

Further, the coordinates of an annotation mark are determined, whichmeans the determination of relative x- and y-coordinates relative to theframe of reference, i.e. the determination of an x-value and a y-valueof a pixel.

In a variant of the method according to one or more example embodimentsof the present invention, the step of detecting new lesions based on thedetected user markings comprises at least one of the following steps:

-   -   performing segmentation task on full DICOM data for the new        lesions,    -   performing detection task on full DICOM data for the new        lesions,    -   performing characterization task on full DICOM data for the new        lesions.

Hence, the automatic detection of new lesions can be improved based onthe user-added lesions.

The captured annotation could be used as a plausibility validator topresent the user only AI findings that he might have missed, e.g.findings for which he has not drawn an annotation already.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is explained below with reference to the figuresenclosed once again. The same components are provided with identicalreference numbers in the various figures.

The figures are usually not to scale.

FIG. 1 shows a schematic view on a combination of a PACS screen and aRIS/reporting screen,

FIG. 2 shows a schematic view on the combination illustrated in FIG. 1 ,wherein screen capture and cross correlation of pixel data is used toidentify which images/series and slice is looked at,

FIG. 3 shows a schematic view on the combination of screens illustratedin FIG. 1 and FIG. 2 , wherein it is detected on which organ the user isfocused, typically based on window levels and widths applied to theimage data in the detected viewports, and wherein an external windowcontaining additional results about the detected image is shown inparallel to these images,

FIG. 4 shows a schematic view on the combination of screens illustratedin FIG. 1 to FIG. 3 , wherein an overlay is generated by a tool, whichis not part of the PACS,

FIG. 5 shows a flow chart diagram illustrating the method for exchangingdata between an external data source for annotation data and anintegrated medical data display system according to an embodiment of thepresent invention,

FIG. 6 shows a schematic view on a medical data analysis system datacomprising an exchange device according to an embodiment of the presentinvention,

FIG. 7 shows a schematic view on a screen of a PACS system, whereinscreen capture is used to track screen content in real-time,

FIG. 8 shows a schematic view on the screen depicted in FIG. 7 , whereina region of least interest is determined in the screen,

FIG. 9 shows a schematic view on the screen depicted in FIG. 7 and FIG.8 , wherein it is determined if the region is suited for display of anexternal content window,

FIG. 10 shows a schematic view on the screen depicted in FIG. 7 to FIG.9 , wherein the external content window is displayed over the suitableregion of least interest,

FIG. 11 shows a schematic view on a screen of a PACS system, wherein aregion of least interest in the screen is determined,

FIG. 12 shows a schematic view on the screen depicted in FIG. 11 ,wherein an external content window is displayed over the suitable regionof least interest,

FIG. 13 shows a flow chart diagram illustrating the method forexchanging data between an external data source for annotation data andan integrated medical data display system according to a secondembodiment of the present invention,

FIG. 14 shows a schematic view on a medical data analysis systemincluding a data exchange device according to a second embodiment of thepresent invention,

FIG. 15 shows a schematic view on a screen of a PACS, wherein areference frame is determined,

FIG. 16 shows a schematic view on the screen depicted in FIG. 15 ,wherein an annotation is detected,

FIG. 17 shows a schematic view on the screen depicted in FIG. 15 or 16 ,wherein an annotation centre is determined and localized,

FIG. 18 shows a flow chart diagram illustrating the method forexchanging data between an external data source for annotation data andan integrated medical data display system according to a thirdembodiment of the present invention, and

FIG. 19 shows a schematic view on a medical data analysis systemcomprising a data exchange device according to a third embodiment of thepresent invention.

DETAILED DESCRIPTION

In FIG. 1 , a schematic view on an integrated medical data displaysystem 1, i.e. a combination 1 of a first screen 2 on the left side,also named image viewer or “PACS monitor”, and a second screen 3 on theright side, also named “RIS/reporting monitor”, is illustrated. TheRIS/reporting monitor 3 is used for displaying mainly non-diagnostictextual information, not containing images, but texts, like patientinformation, reports, worklists, medical record information, etc. TheRIS/reporting monitor 3 is especially used for displaying AI-baseddetermined data. The PACS monitor 2, which is part of a PACS-system(PACS=Picture Archiving and Communication System), comprises a firstviewport which is showing a medical image, for example the lung L of apatient, on the left side. On the right side of the PACS monitor 2, thelung L is shown with another windowing setting (window level and windowwidth). For implementing the data exchange method according to a firstembodiment of the present invention, a tool is used for tracking thescreen of the PACS monitor 2 to capture the screen layout and user inputin the screen of the PACS monitor 2, i.e. the image viewer. Thementioned tool runs on-premises, i.e. on the workstation of the PACSmonitor 2 itself. Screen capture comprises tracking metadata, likecapturing image segments as well as window levels used. In FIG. 1 , thewindow level concerning the illustration of the lung L is detected, forexample. Further, it is detected, for example by OCR or other mechanismsor means, which study or patient is currently opened and displayed onthe first screen 2. Furthermore, the detection is realized by monitoringa particular region of the captured screen layout in which a patient IDor accession number is displayed, or by looking for such a data point inthe captured layout. The second screen on the right side of FIG. 1 ,i.e. the RIS/reporting monitor 3, is not used yet at the stepillustrated in FIG. 1 . Further, an external data source 5 isillustrated in FIG. 1 , which uses an AI-based application for analysingmedical image data received previously and corresponding to the imagedisplayed in the PACS monitor 2 and determines AI-based annotation databased on the received medical image data.

FIG. 2 shows a schematic view on the combination 1 of a PACS monitor 2and a RIS/reporting monitor 3, illustrated in FIG. 1 , wherein screencapture and cross correlation of pixel data is used to identify whichimage, series and slice is looked at. As you can take from FIG. 2 , onthe left side of the screen of the PACS monitor 2, the window displayingthe lung L is segmented by dashed lines and on the left and right sideof the screen of the PACS monitor 2 in the upper right corner, a number“1/270”, also framed by dashed lines, is detected, which is the slicenumber of the image slice displayed on the right side of the screen ofthe PACS monitor 2.

Hence, one viewport on the left side of screen of the PACS monitor 2contains an image and a slice number in the top right corner and oneviewport on the right side contains one image with different windowsetting and a lice number SN in the top right corner, wherein the slicenumber SN can be identical to the first one for synchronized scrolling,or different for unsynchronized scrolling.

The second screen, i.e. the RIS/reporting monitor 3 is not used yet atthe steps illustrated in FIG. 1 and FIG. 2 .

Hence, for each identified image segment IS, it is identified, whichimage slice SN the user is currently visualizing and is eventuallyactive on, either by analysing the metadata displayed in the corners ofthis viewport or by cross correlating the pixel data of the firstscreen, i.e. the PACS monitor 2 with a set of images IM of this patientknown by an external AI-based application of an external data source 5.

As it has been still mentioned, it can be also identified with whichwindow level the user is working for this particular slice, in order toinfer what organs he might be looking at more precisely. Theseoperations, especially the correlations, are envisioned to happen in thecloud or in an edge server, since these locations in an electronicnetwork are the only locations having the “missing part” that has to becorrelated. For example, the PACS monitor 2 shows in an image segment ISa slice with a slice number SN of a series SE of images IM of a study STand the cloud or the edge server, for example the data source 5,possesses results for this image segment IS and slice SN, and possessesa copy of the full image stack (dataset), allowing the correlation to beperformed. In other words, for the image series SE and slice SN andimage segment IS identified a database of an AI-based application of anexternal data source 5 is queried regarding the presence or absence ofresults for this particular image segment IS and slice SN.

In FIG. 3 , a schematic view on the combination 1 of screens 2, 3,illustrated in FIG. 1 and FIG. 2 is shown, wherein based on thedetection which slice, assigned to a special slice number SN, andoptionally which organ, assigned to an image segment IS, is presentlydisplayed, the corresponding AI-based results are shown on the secondscreen 3, i.e. the RIS/reporting monitor. The AI-based results aresupplied from an external data source 5 which stores AI-based determinedannotation data D-EXT.

As you can see in FIG. 3 , a plurality of lesions LS1, LS2, LS3, LS4 aresigned as annotation data D-EXT in the image of the lung L shown on thesecond screen 3. Hence, the available results, and only these resultscorresponding to this slice, are shown in a particular external windowon the second screen 3 that is opened next to the PACS viewer 2, i.e.the first screen 2. This second screen 3 can also be named as “resultreview screen” of an AI application. This window can always stay in theforeground and can maybe only be brought back to background and behidden by an additional button click or toggle that it carries.

For example, results shown in this external window can be presented in amini-viewer. For example, if the radiologist is looking at a chest CT ina particular lung window, the application can present only resultsrelated to the lungs L, for example airways, nodules, parenchyma, andomit results related to bones, aorta and heart.

If the process of adaption of the display of the AI-based information isobtained at sufficient speed, i.e. near real time, the method mightoffer a synchronized scrolling-like functionality for an externalapplication, able to follow the images that a user is looking at in thePACS. If looking at these images, an inference could still be made forthe next findings to show considering, for example, in which directionthe radiologist scrolls through the images. Overall, the near real timebehaviour might be achievable only by using additional techniques, forexample, to reduce the dimensionality of the (captured) image withoutlosing information, for example by down sampling or using anautoencoder-based approach. The streaming of the data in general has tobe technically optimized, which might result in using only an edgeserver and not the cloud.

In FIG. 4 , a schematic view on an alternative scenario of thecombination of screens 2, 3 illustrated in FIG. 1 and FIG. 2 is shown,wherein an overlay OL is generated by a tool, which is not part of thePACS, on the first screen 2. That means that the AI-based results, i.e.the annotations of the lesions LS1, LS2, LS3, LS4, are shown with anoverlay OL generated by the tool, i.e. the data exchange device 60(shown in FIG. 6 ), which overlay OL is not part of the PACS, but isimplemented as a “fake” overlay, ensuring editability “in” the PACSmonitor 2 and interaction with the result without a deeper PACSintegration. In other words, an overlay OL on top of the slice shown inthe PACS monitor 2 is generated by the data exchange device 60,corresponding to the result, that would enable real-time interactionbetween the result object “in context”, though the result object is nota part of the PACS itself. The overlay OL can be generated in a way tomaximize usability, for example using appropriate colours and stylesthat make the result well distinguishable from the original image. Ifthe overlay OL is interactive, then it needs to have the focus on theoverlay OL also while scrolling in the PACS. This might be solvable byadapting the size of the overlay OL, so that scrolling in the backgroundapplication is enabled, or by implementing an event-forwarding logic tothe background program.

FIG. 5 shows a flow chart diagram 500 illustrating the method forexchanging data between an external data source 5 for annotation dataand an integrated medical data display system 1 according to a firstembodiment of the present invention, wherein an adaption of data of anexternal data source to an integrated medical data display system iscarried out.

In step 5.I, a tool CS-T is used, which is capable of screen tracking tocapture screen content SCR-C in real-time and to capture user input UIon a screen 2 of a PACS system.

In step 5.II it is detected, for example by OCR, which study ST orpatient P is currently opened on the screen 2. The detection is realizedby monitoring a particular region of the captured screen layout, inwhich a patient ID P-ID and/or an accession number ACN is displayed, orby extracting these data points from the full captured screen byapplying additional image processing techniques.

In step 5.III, it is classified based on the captured screen layout,which screen portions SCR-P correspond to which image segments IS. Forexample, (x,y)-coordinates of bounding boxes defining viewports aredetermined.

In step 5.IV, it is identified for each identified image segments IS,which image IM the user is currently visualizing and is eventuallyactive on. This is done either by analysing the metadata displayed inthe corners of this viewport or by cross correlating the pixel data witha set of images of this patient known by an external application. Forexample, the image IM is identified as an image of the lung of thepatient P. Further, also the slice number SN as well as a series numberSE of a study ST are identified.

In step 5.V, data D-EXT, for example annotation data, from an externaldata source 5 are selected, which are assigned to the determinedinformation, for example an image IM of the lung. The external dataD-EXT comprises AI results and the step of selecting the external dataD-EXT from the external data source 5 comprises searching the externaldata source 5 for AI results which correspond to the determined content.For example, AI results are requested which are identified by the imageseries SE, slice SN and study ST.

In step 5.VI, the selected data D-EXT are displayed complementary to thedisplayed information such that the selected data do not conceal thedisplayed information.

For the complementary display, as shown in FIG. 4 , an overlay OL on topof the medical image, shown on the PACS monitor 2, but independently ofthe display system of the PACS can be used. The medical image datadisplayed on the PACS monitor 2 can be understood as a first applicationwindow. Then, the overlay OL is realised as a second application windowwhich is at least partly transparent. The second application window ispresented in foreground overlapping with the PACS application window onthe PACS monitor 2. The overlay is registered with the medical image ofthe PACS such that the results depicted by the overlay are located atcorresponding positions of the displayed medical image. Hence, theregistration comprises an adaption of the position and the size of theoverlay OL to the underlying image. The overlay is generated in a way tomaximize usability using appropriate colours or styles that make theresult well distinguishable from the original image. Another(less-integrate) option, as stated in FIG. 3 , is to display results ina separate window that is not overlapping with the detected images.

In FIG. 6 , a schematic view on a medical data analysis system 10, i.e.an arrangement comprising a data exchange device 60 according to anembodiment of the present invention, an integrated medical data displaysystem 1 and an evaluation unit 15 with an external data source 5, isillustrated. The data exchange device 60 comprises a determination unit61 for determining information displayed on a screen 2 of an integratedmedical data display system 1 by tracking the screen 2.

The determination unit 61 comprises a screen capture sub-unit 61 a whichis capable of screen tracking to capture screen content SCR-C inreal-time and to capture user input UI.

Further, the determination unit 61 comprises a text recognition sub-unit61 b which detects by OCR or other text recognition techniques, whichstudy ST or patient P is currently opened on the screen. The detectionis realised by monitoring a particular region of the captured screenlayout in which a patient ID P-ID and/or accession number ACN (not shownin FIG. 6 ) is displayed, or by extracting these data points from thefull captured screen.

Furthermore, the determination unit 61 comprises a classification unit61 c which classifies based on the captured screen layout, which screenportions SCR-P correspond to which image segments IS. For example, (x,y)-coordinates of bounding boxes defining viewports are determined.

The determination unit 61 also comprises an identification unit 61 dwhich identifies for each identified image segments IS which image IMthe user is currently visualizing and is eventually active on. This isdone either by analysing the metadata displayed in the corners of aviewport on a screen or by cross correlating the pixel data with a setof images of this patient known by an external application which issymbolised in FIG. 6 by an evaluation unit 15, applied on an externaldata source 5. For example, the image IM is identified as an image ofthe lung L of the patient P, with slice number SN and series SE.

The data exchange device 60 also comprises a selection unit 62 forselecting or filtering data D-EXT from the external data source 5assigned to the determined information IM.

The data exchange device 60 then sends the selected data D-EXT to theintegrated medical data display system 1 for displaying the selecteddata complementary to the displayed information. For example, theselected data D-EXT are shown on a PACS viewer 2 as an overlay OL, asdepicted in FIG. 4 or on a separated second screen 3 as depicted in FIG.3 .

The data exchange device 60 can be a (personal) computer, a workstation,a virtual machine running on host hardware, a microcontroller, or anintegrated circuit. In particular, the data exchange device 60 can be amobile device, e.g., a smartphone or a tablet. As an alternative, thedata exchange device 60 can be a real or a virtual group of computers(the technical term for a real group of computers is “cluster”, thetechnical term for a virtual group of computers is “cloud”).

In general, the determination unit 61, the selection unit 62 and theextraction unit 63 can comprise hardware elements and software elements,for example a microprocessor, a CPU (acronym for “central processingunit”), a GPU (acronym for “graphical processing unit”), a fieldprogrammable gate array (an acronym is “FPGA”) or an ASIC (acronym for“application-specific integrated circuit”). The determination unit 61,the selection unit 62 and the extraction unit 63 can be configured formultithreading, i.e., the determination unit 61, the selection unit 62and the extraction unit 63 can host different computation processes atthe same time, executing them either in parallel or switching betweenactive and passive computation processes. In particular, thedetermination unit 61, the selection unit 62 and the extraction unit 63can be a combination of several hardware and software elements, inparticular, the determination unit 61, the selection unit and theextraction unit 63 can comprise one or more determination unitsrespectively selection units respectively extraction units assubcomponents. FIG. 7 shows a schematic view on a screen 2 of a PACSsystem, wherein screen capture is used to track screen content inreal-time. FIG. 7 is referred to a second embodiment of the presentinvention, which relates to an appropriate positioning of content of anexternal data source 5 (not shown in FIG. 7 ) on a screen 2 of a PACSsystem. For example, the screen 2 in FIG. 7 comprises four viewports andin the upper right corner a field 2 b for displaying metadata like, inthis example, image thumbnails.

FIG. 8 shows a schematic view on the screen 2 depicted in FIG. 7 ,wherein a region 2 c of least interest is determined in the screen 2.Based on the information captured from the screen 2 by screen capture,corner locations and borders are detected. This can be done via alearning-based approach. Another alternative can be based onconvolutional processing. There are for example well-known specifickernels that enable the detection of horizontal and vertical lines ofdifferent width, which can be used by thresholding to retrieve thedifferent viewports and areas on the screen 2. The detection of regions2 c of least interest can also be realized interactive. For example, afirst screenshot can be used for capturing the overall structure of theused screen 2 and the user is requested, using some user interface, todefine regions on this image that could be considered as a region 2 c ofleast interest. The request can be done in advance for a set of commonlayouts that will then be persisted.

In FIG. 9 , a schematic view on the screen 2 depicted in FIG. 7 and FIG.8 is shown, wherein it is determined if the region 2 c of least interestis suited for displaying an external content window ECW (depicted inFIG. 10 ). Eventually, a resizing is performed to fit this space anddisplay correctly. A successful resizing can be achieved easily with aresponsive design of the application and with a resizing of therespective container window. If several candidate positions areavailable, the optimal position can be chosen based on the best size fitthough, or based on a ranking of the different candidates relying ondiverse metrics like aspect-ratio fit of the window to display, contenttype that would be hidden if placing a window at a particular position,etc.

FIG. 10 shows a schematic view on the screen 2 depicted in FIG. 7 toFIG. 9 , wherein the external content window ECW is displayed over thesuitable region 2 c of least interest.

In FIG. 11 , a schematic view on a screen 2 of a PACS system is shown,wherein a region 2 c of least interest in the screen 2 is determined,which extents additionally onto a neighboured viewport.

In FIG. 12 , a schematic view on the screen 2 depicted in FIG. 11 , isillustrated, wherein an external content window ECW is displayed overthe suitable region 2 c of least interest, which is extended enough toshow an annotated image of a currently depicted sectional head image.

FIG. 13 shows a flow chart diagram 1300 illustrating the method forexchanging data between an external data source for annotation data andan integrated medical data display system according to a secondembodiment of the present invention.

In step 13.I, a screen capture method is used to track screen contentSCR-C in real-time. Screen capture is done for detecting an appropriateposition for displaying content of the mentioned external data source ona screen 2 of a PACS system (shown in FIG. 7 to FIG. 12 ).

In step 13.II, a region 2 c of least interest is determined in thescreen 2. Based on the information captured from the screen 2 by screencapture, corner locations and borders are detected. This can be done viaa learning-based approach.

In step 13.III, it is determined if the region 2 c of least interest issuited for display of an external content window ECW. If that is thecase, which is symbolized with “y”, the method continues with step13.VII. In case the size does not suit, which is symbolized with “n”, itis determined in step IV, if resizing RES is possible. If resizing RESis possible, which is symbolized in FIG. 13 with “Y”, a resizing isperformed in step 13.V to fit this space and display correctly.

If resizing RES is not possible or feasible, which is symbolized in FIG.13 with “n”, e.g. because the resulting resized window would be toosmall or below a certain threshold size, too distorted, etc., then, instep 13.VI, the decision can be taken automatically to display thecontent on a separate monitor, by explicitly hiding content, whileacknowledging that better readability would be obtained for the contentdisplayed in the external window.

In step 13.VII, an external content window ECW is displayed over thesuitable region 2 c of least interest or on a separate monitor dependingon the decisions of step 13.III and 13.IV.

In FIG. 14 , a schematic view on a medical data analysis system 10,comprising a data exchange device 60 and an integrated medical datadisplay system 1 according to a second embodiment of the presentinvention is illustrated.

The data exchange device 60 comprises a determination unit 61 fordetermining information displayed on a screen 2 (not shown) of theintegrated medical data display system 1 by tracking the screen 2.

The determination unit 61 comprises a screen capture sub-unit 61 a whichis capable of screen tracking to capture screen content SCR-C inreal-time.

Further, the determination unit 61 comprises a location detection unit61 dd for detecting a region 2 c of least interest in the screen. Basedon the information captured from the screen 2 by screen capture, cornerlocations and borders are detected. This can be done via alearning-based approach.

The data exchange device 60 also comprises a selection unit 62 forselecting data D-EXT from the external data source 5 for displayingthese data D-EXT in an external content window ECW. The selection unit62 comprises a selection sub-unit 62 a for retrieving data D-EXT fromthe external data source 5 and a formation unit 62 b for resizing theretrieved data D-EXT such that it suits the region 2 c of least interestand for positioning an external content window ECW comprising theretrieved data D-EXT in the region 2 c of least interest.

Further, the external content window ECW is transmitted to theintegrated medical data display system 1 for displaying the externalcontent window ECW at the determined region 2 c.

Alternatively, only the data necessary to position and display theexternal content window ECW correctly are transmitted. For example, notthe full window ECW itself is transmitted, but only some parameters like“ECW corner positions”, “ECW width”, “ECW height”, “ECW centre”, etc.are transmitted so that the window ECW knows, how to position itselfbased on these received parameters.

FIG. 15 shows a schematic view on a screen 2 of a PACS, wherein areference frame 2 d is determined on the screen 2. In the thirdembodiment, a user input into the image viewer 2 which is an annotatedlesion LS is captured by a screen tracking tool on the screen 2 in theupper right viewport determined by the reference frame 2 d. Alsoadditional viewports 2 a are depicted in FIG. 15 , which are notselected.

For this object, a frame 2 d of reference is autodetected. For thispurpose, the workflow is usually sufficiently standardized topreconfigure a particular window on a split screen that is always usedto annotate series. Alternatively, a particular window can also beautodetected by for example analysing displayed metadata like seriesdescriptions which are usually displayed as overlay on the images.

FIG. 16 shows a schematic view on the screen 2 depicted in FIG. 15 ,wherein an annotation ANO is detected in the detected frame 2 d.Detection of user markings, i.e. annotations ANO, can be realized basedon rgb values. Alternatively, image content can be subtracted from theunannotated image to see largest deviation and therefore compute a maskof the annotation ANO. Annotations ANO can be of different types, forexample, but not limited to lines, simple or perpendicular, rectangular,circular or ellipsoidal bounding boxes (regions of interest) orcrosshairs.

FIG. 17 shows a schematic view on the screen 2 depicted in FIG. 15 or 16, wherein an annotation centre AC is determined and localized. Forexample coordinates CO of the annotation mark AN are determined.Depending on the annotation types, it may be useful to determine andpersist more than just the centre of an annotation, for example todetermine more than one coordinate point per annotation. For example,for a crosshair, the centre can be determined, but also the maximumdiameter or the coordinates of the two lines making of the crosshair.

For a circular annotation, as another example, the centre of the circleand an approximate radius or diameter of the circle can be determined.The determination of the approximate location of the annotation usingthe centre is, however, the minimum requirement and that can besufficient for some application. Further, for every captured mark ANO,annotation identifier AN, corresponding slice number SN andcorresponding x- and y-coordinates CO in pixels are saved and sent to aremote location. This new captured annotation data might be transmittedencrypted so that the gained knowledge is not exposed to third-partysystems without explicit permission.

FIG. 18 shows a flow chart diagram illustrating the method forexchanging data between an external data source and an integratedmedical data display system according to a third embodiment of thepresent invention.

In step 18.I, a screen capture tool is used to track screen contentSCR-C in real-time. Screen capture is done for detecting a referenceframe 2 d.

In step 18.II, an annotation ANO is detected in the detected frame 2 d.As mentioned-above, detection of user markings, i.e. annotations ANO canbe realised based on rgb values. Alternatively, image content can besubtracted from unannotated image to see largest deviation and thereforecompute a mask of the annotation ANO.

In step 18.III an annotation centre AC is determined and localized.Further, coordinates CO of the annotation mark ANO are determined.Furthermore, for every captured mark AN, annotation identifier AN,corresponding slice number SN and corresponding x- and y-coordinates COin pixels are saved and sent to a remote location.

FIG. 19 shows a schematic view on a medical data analysis system 10comprising a data exchange device 60 according to a third embodiment ofthe present invention.

The data exchange device 60 comprises a determination unit 61 fordetermining information SCR-C displayed on a screen 2 of an integratedmedical data display system 1 by tracking the screen 2.

The determination unit 61 comprises a screen capture sub-unit 61 a whichis capable of screen tracking to capture screen content SCR-C inreal-time.

Further, the determination unit 61 comprises a reference frame detectionunit 61 f for detecting a reference frame 2 d based on the capturedscreen content SCR-C.

Furthermore, the determination unit 61 comprises an annotation detectionunit 61 g for detecting annotations ANO in the detected frame 2 d.

Furthermore, the determination unit 61 comprises an annotation centredetection unit 61 h for determining an annotation centre AC andlocalizing the annotation centre AC. Depending on annotation type, alsoother types of landmarks can be determined, for example“centre+extremities” or “centre+radius”.

The data exchange device 60 also comprises an annotation addition unit63 for determining for every annotation ANO an annotation number AN,corresponding slice number SN and corresponding x- and y-coordinates COin pixels and saving the additional annotation data ANO in an externaldata source 5.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,layers, and/or sections, these elements, components, regions, layers,and/or sections, should not be limited by these terms. These terms areonly used to distinguish one element from another. For example, a firstelement could be termed a second element, and, similarly, a secondelement could be termed a first element, without departing from thescope of example embodiments. As used herein, the term “and/or,”includes any and all combinations of one or more of the associatedlisted items. The phrase “at least one of” has the same meaning as“and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or operation in addition tothe orientation depicted in the figures. For example, if the device inthe figures is turned over, elements described as “below,” “beneath,” or“under,” other elements or features would then be oriented “above” theother elements or features. Thus, the example terms “below” and “under”may encompass both an orientation of above and below. The device may beotherwise oriented (rotated 90 degrees or at other orientations) and thespatially relative descriptors used herein interpreted accordingly. Inaddition, when an element is referred to as being “between” twoelements, the element may be the only element between the two elements,or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including “on,”“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Incontrast, when an element is referred to as being “directly” on,connected, engaged, interfaced, or coupled to another element, there areno intervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments. As used herein, the singular forms “a,” “an,” and “the,”are intended to include the plural forms as well, unless the contextclearly indicates otherwise. As used herein, the terms “and/or” and “atleast one of” include any and all combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises,” “comprising,” “includes,” and/or “including,” when usedherein, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof. As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items. Expressions such as “at least one of,” whenpreceding a list of elements, modify the entire list of elements and donot modify the individual elements of the list. Also, the term “example”is intended to refer to an example or illustration.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, e.g., those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

It is noted that some example embodiments may be described withreference to acts and symbolic representations of operations (e.g., inthe form of flow charts, flow diagrams, data flow diagrams, structurediagrams, block diagrams, etc.) that may be implemented in conjunctionwith units and/or devices discussed above. Although discussed in aparticular manner, a function or operation specified in a specific blockmay be performed differently from the flow specified in a flowchart,flow diagram, etc. For example, functions or operations illustrated asbeing performed serially in two consecutive blocks may actually beperformed simultaneously, or in some cases be performed in reverseorder. Although the flowcharts describe the operations as sequentialprocesses, many of the operations may be performed in parallel,concurrently or simultaneously. In addition, the order of operations maybe re-arranged. The processes may be terminated when their operationsare completed, but may also have additional steps not included in thefigure. The processes may correspond to methods, functions, procedures,subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments. Thepresent invention may, however, be embodied in many alternate forms andshould not be construed as limited to only the embodiments set forthherein.

In addition, or alternative, to that discussed above, units and/ordevices according to one or more example embodiments may be implementedusing hardware, software, and/or a combination thereof. For example,hardware devices may be implemented using processing circuitry such as,but not limited to, a processor, Central Processing Unit (CPU), acontroller, an arithmetic logic unit (ALU), a digital signal processor,a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC), a programmable logic unit, a microprocessor, orany other device capable of responding to and executing instructions ina defined manner. Portions of the example embodiments and correspondingdetailed description may be presented in terms of software, oralgorithms and symbolic representations of operation on data bits withina computer memory. These descriptions and representations are the onesby which those of ordinary skill in the art effectively convey thesubstance of their work to others of ordinary skill in the art. Analgorithm, as the term is used here, and as it is used generally, isconceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of optical, electrical, or magnetic signals capable of beingstored, transferred, combined, compared, and otherwise manipulated. Ithas proven convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like.

It should be borne in mind that all of these and similar terms are to beassociated with the appropriate physical quantities and are merelyconvenient labels applied to these quantities. Unless specificallystated otherwise, or as is apparent from the discussion, terms such as“processing” or “computing” or “calculating” or “determining” of“displaying” or the like, refer to the action and processes of acomputer system, or similar electronic computing device/hardware, thatmanipulates and transforms data represented as physical, electronicquantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

In this application, including the definitions below, the term ‘module’or the term ‘controller’ may be replaced with the term ‘circuit.’ Theterm ‘module’ may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

Software may include a computer program, program code, instructions, orsome combination thereof, for independently or collectively instructingor configuring a hardware device to operate as desired. The computerprogram and/or program code may include program or computer-readableinstructions, software components, software modules, data files, datastructures, and/or the like, capable of being implemented by one or morehardware devices, such as one or more of the hardware devices mentionedabove. Examples of program code include both machine code produced by acompiler and higher level program code that is executed using aninterpreter.

For example, when a hardware device is a computer processing device(e.g., a processor, Central Processing Unit (CPU), a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a microprocessor, etc.), the computer processing devicemay be configured to carry out program code by performing arithmetical,logical, and input/output operations, according to the program code.Once the program code is loaded into a computer processing device, thecomputer processing device may be programmed to perform the programcode, thereby transforming the computer processing device into a specialpurpose computer processing device. In a more specific example, when theprogram code is loaded into a processor, the processor becomesprogrammed to perform the program code and operations correspondingthereto, thereby transforming the processor into a special purposeprocessor.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical or virtual equipment, or computerstorage medium or device, capable of providing instructions or data to,or being interpreted by, a hardware device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, forexample, software and data may be stored by one or more computerreadable recording mediums, including the tangible or non-transitorycomputer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the formof a program or software. The program or software may be stored on anon-transitory computer readable medium and is adapted to perform anyone of the aforementioned methods when run on a computer device (adevice including a processor). Thus, the non-transitory, tangiblecomputer readable medium, is adapted to store information and is adaptedto interact with a data processing facility or computer device toexecute the program of any of the above mentioned embodiments and/or toperform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolicrepresentations of operations (e.g., in the form of flow charts, flowdiagrams, data flow diagrams, structure diagrams, block diagrams, etc.)that may be implemented in conjunction with units and/or devicesdiscussed in more detail below. Although discussed in a particularmanner, a function or operation specified in a specific block may beperformed differently from the flow specified in a flowchart, flowdiagram, etc. For example, functions or operations illustrated as beingperformed serially in two consecutive blocks may actually be performedsimultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processingdevices may be described as including various functional units thatperform various operations and/or functions to increase the clarity ofthe description. However, computer processing devices are not intendedto be limited to these functional units. For example, in one or moreexample embodiments, the various operations and/or functions of thefunctional units may be performed by other ones of the functional units.Further, the computer processing devices may perform the operationsand/or functions of the various functional units without sub-dividingthe operations and/or functions of the computer processing units intothese various functional units.

Units and/or devices according to one or more example embodiments mayalso include one or more storage devices. The one or more storagedevices may be tangible or non-transitory computer-readable storagemedia, such as random access memory (RAM), read only memory (ROM), apermanent mass storage device (such as a disk drive), solid state (e.g.,NAND flash) device, and/or any other like data storage mechanism capableof storing and recording data. The one or more storage devices may beconfigured to store computer programs, program code, instructions, orsome combination thereof, for one or more operating systems and/or forimplementing the example embodiments described herein. The computerprograms, program code, instructions, or some combination thereof, mayalso be loaded from a separate computer readable storage medium into theone or more storage devices and/or one or more computer processingdevices using a drive mechanism. Such separate computer readable storagemedium may include a Universal Serial Bus (USB) flash drive, a memorystick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other likecomputer readable storage media. The computer programs, program code,instructions, or some combination thereof, may be loaded into the one ormore storage devices and/or the one or more computer processing devicesfrom a remote data storage device via a network interface, rather thanvia a local computer readable storage medium. Additionally, the computerprograms, program code, instructions, or some combination thereof, maybe loaded into the one or more storage devices and/or the one or moreprocessors from a remote computing system that is configured to transferand/or distribute the computer programs, program code, instructions, orsome combination thereof, over a network. The remote computing systemmay transfer and/or distribute the computer programs, program code,instructions, or some combination thereof, via a wired interface, an airinterface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices,and/or the computer programs, program code, instructions, or somecombination thereof, may be specially designed and constructed for thepurposes of the example embodiments, or they may be known devices thatare altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run anoperating system (OS) and one or more software applications that run onthe OS. The computer processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For simplicity, one or more example embodiments may beexemplified as a computer processing device or processor; however, oneskilled in the art will appreciate that a hardware device may includemultiple processing elements or processors and multiple types ofprocessing elements or processors. For example, a hardware device mayinclude multiple processors or a processor and a controller. Inaddition, other processing configurations are possible, such as parallelprocessors.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium (memory).The computer programs may also include or rely on stored data. Thecomputer programs may encompass a basic input/output system (BIOS) thatinteracts with hardware of the special purpose computer, device driversthat interact with particular devices of the special purpose computer,one or more operating systems, user applications, background services,background applications, etc. As such, the one or more processors may beconfigured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language) or XML (extensible markuplanguage), (ii) assembly code, (iii) object code generated from sourcecode by a compiler, (iv) source code for execution by an interpreter,(v) source code for compilation and execution by a just-in-timecompiler, etc. As examples only, source code may be written using syntaxfrom languages including C, C++, C#, Objective-C, Haskell, Go, SQL, R,Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5,Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang,Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one example embodiment relates to the non-transitorycomputer-readable storage medium including electronically readablecontrol information (processor executable instructions) stored thereon,configured in such that when the storage medium is used in a controllerof a device, at least one embodiment of the method may be carried out.

The computer readable medium or storage medium may be a built-in mediuminstalled inside a computer device main body or a removable mediumarranged so that it can be separated from the computer device main body.The term computer-readable medium, as used herein, does not encompasstransitory electrical or electromagnetic signals propagating through amedium (such as on a carrier wave); the term computer-readable medium istherefore considered tangible and non-transitory. Non-limiting examplesof the non-transitory computer-readable medium include, but are notlimited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of the non-transitory computer-readable medium include, but arenot limited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

Although described with reference to specific examples and drawings,modifications, additions and substitutions of example embodiments may bevariously made according to the description by those of ordinary skillin the art. For example, the described techniques may be performed in anorder different with that of the methods described, and/or componentssuch as the described system, architecture, devices, circuit, and thelike, may be connected or combined to be different from theabove-described methods, or results may be appropriately achieved byother components or equivalents.

The above descriptions are merely preferred embodiments of the presentdisclosure, but not intended to limit the present disclosure, and anymodifications, equivalent replacements, improvements, etc. made withinthe spirit and principle of the present disclosure, should be includedwithin the scope of protection of the present disclosure.

Further, the use of the undefined article “a” or “one” does not excludethat the referred features can also be present several times. Likewise,the term “unit” or “device” does not exclude that it consists of severalcomponents, which may also be spatially distributed.

What is claimed is:
 1. A method for exchanging data between an external data source for annotations and an integrated medical data display system, the method comprising: determining information displayed on a screen of the integrated medical data display system by capturing the screen; and performing at least one of selecting data from the external data source assigned to the information displayed on the screen, and displaying the selected data as complementary to the information displayed on the screen, or extracting annotations from the integrated medical data display system based on the information displayed on the screen, and appending the annotations to the external data source for annotations.
 2. The method according to claim 1, wherein the information displayed on the screen comprises at least one of: content of medical data, formal information about a format of displayed data, or non-image information including patient metadata or examination metadata.
 3. The method according to claim 2, wherein the information displayed on the screen includes the formal information about the format of displayed data, and the formal information includes at least one of a screen layout including a screen portion, the screen portion including at least one of a viewport or a delimited screen portion including an image segment, or user input to the integrated medical data display system.
 4. The method according to claim 3, wherein the determining information displayed on a screen comprises at least one of: using a tool capable of screen tracking to capture screen content, detecting which study or patient is currently displayed on the screen, classifying, based on the screen layout, which screen portion corresponds to which image segment, and identifying, for each classified image segment, which image a user is at least one of currently visualizing or active on, or identifying a window level the user is working with for a particular slice.
 5. The method according to claim 1, wherein the external data source includes results, and the selecting data from the external data source includes searching the external data source for results corresponding to the information displayed on the screen, wherein the selected data includes the results.
 6. The method according to claim 4, wherein the displaying the selected data comprises: generating an overlay on the information displayed on the screen of the integrated medical data display system, wherein the overlay includes annotations corresponding to the information displayed on the screen, and the overlay is registered with the information displayed on the screen.
 7. The method according to claim 1, wherein the displaying the selected data comprises: showing the selected data in a particular external window that is opened next to the integrated medical data display system, or at least partially overlapping the selected data with the integrated medical data display system.
 8. The method according to claim 3, wherein the determining information displayed on a screen comprises: determining corner locations or borders of displayed viewports on the screen based on the screen layout.
 9. The method according to claim 3, wherein the determining information displayed on a screen comprises at least one of: determining at least one active viewport using mouse activity tracked using screen capture, or determining a screen portion not containing an image segment, and displaying an external application window within the screen portion not containing the image segment.
 10. The method according to claim 8, wherein the determining information displayed on a screen comprises: using a metric to determine at least one screen portion as a monitor segment of least-interest based on the information displayed on the screen, and displaying an external application window within the monitor segment of least-interest.
 11. The method according to claim 10, wherein in case a plurality of monitor segments of least-interest are determined, a position for the external application window is chosen at least one of based on a best size fit between the external application window and the plurality of monitor segments of least-interest, based on an aspect ratio fit between the external application window and the plurality of monitor segments of least-interest, or overlapping the external application window with images of the integrated medical data display system, in response to no monitor segment of least interest that fits the external application window being found.
 12. The method according to claim 1, wherein the determining information displayed on the screen of the integrated medical data display system comprises: detecting a reference frame based on the information displayed on the screen, and detecting annotations in the reference frame based on the information displayed on the screen.
 13. A data exchange device for exchanging data between an external data source for annotations and an integrated medical data display system, the data exchange device comprising: a determination unit configured to determine information displayed on a screen of the integrated medical data display system by capturing the screen; and at least one of a selection unit or an extraction unit, wherein the selection unit is configured to select data from the external data source assigned to the information displayed on the screen, and display the selected data as complementary to the information displayed on the screen, and the extraction unit is configured to extract annotations from the integrated medical data display system based on the information displayed on the screen, and append the extracted annotations to the external data source for annotations.
 14. A non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to carry out the method of claim
 1. 15. A data exchange device for exchanging data between an external data source for annotations and an integrated medical data display system, the data exchange device comprising: a memory storing computer executable instructions; and at least one processor configured to execute the computer executable instructions to cause the data exchange device to determine information displayed on a screen of the integrated medical data display system by capturing the screen, and at least one of select data from the external data source assigned to the information displayed on the screen, and display the selected data as complementary to the information displayed on the screen, or extract annotations from the integrated medical data display system based on the information displayed on the screen, and append the extracted annotations to the external data source for annotations.
 16. The method according to claim 3, wherein the determining information displayed on a screen comprises at least one of: using a tool capable of screen tracking to capture screen content, detecting which study or patient is currently displayed on the screen, classifying, based on the screen layout, which screen portion corresponds to which image segment, and identifying, for each classified image segment, which image a user is at least one of currently visualizing or active on, or identifying a window level the user is working with for a particular slice, and inferring organs being reviewed based on the window level.
 17. The method according to claim 4, wherein the external data source includes results, and the selecting data from the external data source includes searching the external data source for results corresponding to the information displayed on the screen, wherein the selected data includes the results.
 18. The method according to claim 17, wherein the displaying the selected data comprises: generating an overlay on the information displayed on the screen of the integrated medical data display system, wherein the overlay includes annotations corresponding to the information displayed on the screen, and the overlay is registered with the information displayed on the screen.
 19. The method according to claim 4, wherein the determining information displayed on a screen comprises: determining corner locations or borders of displayed viewports on the screen based on the screen layout.
 20. The method according to claim 4, wherein the determining information displayed on a screen comprises at least one of: determining at least one active viewport using mouse activity tracked using screen capture, or determining a screen portion not containing an image segment, and displaying an external application window within the screen portion not containing the image segment. 